Second Reader

ABSTRACT

The present invention relates to a method and system that automatically determines malignancy in mammograms in parallel with a human operator. More particularly, the present invention relates to providing a reliable automated malignancy determination in parallel to a human operator to reduce the need for two human operators in a mammography analysis workflow.Aspects and/or embodiments seek to provide a method of automatically assessing mammography data in parallel with a human operator. Aspects and/or embodiments also seek to address the problems relating to providing a substantially reliable second reader to allow a single operator to analyse and diagnose mammography data.

FIELD

The present invention relates to a method and system that substantiallyautomatically determines malignancy in mammograms in parallel with ahuman operator. More particularly, the present invention relates toproviding a reliable automated malignancy determination in parallel to ahuman operator to reduce the need for two human operators in amammography analysis workflow.

BACKGROUND

Mammography is a medical imaging modality widely used for breast cancerdetection. Mammography makes use of “soft” X-rays to produce detailedimages of the internal structure of the human breast. These images arecalled mammograms and use of mammography is considered to be the goldstandard in the early detection of breast abnormalities (which canprovide a valid diagnosis of a cancer in a curable phase).

Unfortunately, the procedure of analysing mammograms is oftenchallenging. The density and tissue type of the breasts are highlyvaried and in turn present a high variety of visual features due topatient genetics. These background visual patterns can obscure theoften-tiny signs of malignancies which may then be easily overlooked bythe human eye. Thus, the analyses of mammograms often lead tofalse-positive or false-negative diagnostic results which may causemissed treatment (in the case of false-negatives) as well as unwantedpsychological and sub-optimal downstream diagnostic and treatmentconsequences (in the case of false-positives).

Most developed countries maintain a population-wide screening program,i.e. a comprehensive system for calling in women of a certain age group,free of symptoms, to have regular breast screening. These screeningprograms require highly standardised protocols to be followed by trainedand experienced specialist doctors who can reliably analyse a largenumber of mammograms routinely. Most professional guidelines stronglysuggest the reading of each mammogram by two equally expert radiologists(an approach known as double-reading). Nowadays, with the number ofavailable highly skilled radiologists being scarce, and decreasing, thedouble-reading approach is often impractical or impossible.

The involvement of two expert radiologists significantly increases thecost of each case and also prolongs the time for a patient to receivethe results of a scan. In some cases, a suspicious lesion may even bemissed by both expert radiologists.

Therefore, there is a need to improve, if not at least maintain, thequality of mammography results whilst adhering to guidelines thatrequire or strongly suggest a double-read process.

SUMMARY OF THE INVENTION

Aspects and/or embodiments seek to provide a method of substantiallyautomatically assessing mammography data in parallel with a humanoperator. Aspects and/or embodiments also seek to address the problemsrelating to providing a substantially reliable second reader to allow asingle operator to analyse and diagnose mammography data.

According to a first aspect, there is provided a computer-aided methodof analysing mammographic images, the method comprising the steps ofreceiving a plurality of mammograms (10); performing a first analysis(30) on the plurality of mammograms (10) comprising identifying amalignancy classification for each of the mammograms; determining amalignancy output value (30Y) for each of mammograms dependent upon thefirst analysis (30); determining an average malignancy output value byaveraging the malignancy output values for the plurality of mammograms;thresholding the average malignancy output value to generate an outputbinary malignancy value (60Y); performing a second analysis (40) on theplurality of mammograms to determine a plurality of localisation dataparameters (40X) for each mammogram; and generating (70) outputlocalisation data for the plurality of mammograms in dependence upon theoutput binary malignancy value.

In this way, the need for an additional highly skilled medicalprofessional to perform a second reading of the mammogram can beeliminated and the second read can be performed automatically, andsubstantially instantaneously. The method may also reduce the risk ofhuman error.

By receiving multiple input images, radiologists can perform case-wiseanalysis for patients and substantially determine the likelihood of amalignant lesion after analysing multiple mammographic views. In orderto combat the generation of multiple false-positives that can limit theeffectiveness/reliability of current machine learning based methods, themethod may only provide localisation data for lesions if the analysisfrom the case-wise review of the mammograms suggest there is a malignantlesion.

Optionally, there can be performed the further step of pre-processing aplurality of mammograms to improve malignancy classification for each ofthe mammograms, the step of pre-processing further comprising the use ofone or more trained neural networks. Primarily, convolutional neuralnetworks can be used but other types may be used, such as capsulenetworks.

Optionally, the step of performing the first analysis on the pluralityof mammograms is conducted using one or more trained convolutionalneural network classifier. As an example, the convolutional neuralnetworks (or CNNs) can be ConvNets.

Optionally, the weights of the trained convolutional neural networkclassifier are frozen in dependence upon data used to train theconvolutional neural network classifier.

Optionally, the plurality of mammograms comprises: a left side cranialcaudal mammogram, L-CC; a right side cranial caudal mammogram, R-CC; aleft side medio-lateral-oblique mammogram, L-MLO; and a right sidemedio-lateral-oblique mammogram, R-MLO.

Optionally, the average malignancy output value comprises anycombination of; an average value for all L-CC malignancy output values;an average value for all R-CC malignancy output values; an average valuefor all L-MLO malignancy output values; an average value for all R-MLOmalignancy output values; an average value for all left-side mammogrammalignancy output values; and an average value for all right-sidemammogram malignancy output values.

Optionally, a max operator is performed between the average value forall left-side mammogram malignancy output values and the average valuefor all right-side mammogram malignancy output values to determine theaverage malignancy output value.

Optionally, the step of performing a second analysis on the plurality ofmammograms is conducted using one or more trained regional convolutionalneural network, RCNN.

Optionally, the one or more trained RCNNs comprises a plurality ofsub-divisional networks to determine the plurality of localisation dataparameters, the sub-divisional networks provide any one or combinationof: a bounding box generation model; a segmentation model; and amalignancy classification type model.

Optionally, the one or more RCNNs are coupled to an output convolutionallayer of the one or more convolutional neural network used to performthe first analysis. Optionally, the one or more RCNNs are trained usingthe weights of the of the one or more convolutional neural network usedto perform the first analysis.

Optionally, the one or more trained RCNNs generates an overlay maskindicating a lesion of interest, the mask further comprises a malignancyprobability value.

Optionally, the bounding box generation model generates bounding boxregression with none-max suppression in order to locate lesions ofinterest.

Optionally, the segmentation model provides a segmentation outlines ofanatomical regions and/or lesions, the segmentation model furthercomprises localisational characteristics.

Optionally, the malignancy classification type model identifies a tissuetype and density category classification for the breast.

Optionally, the sub-divisional networks comprise an ensemble of maskscreated by the sub-division networks. Optionally, the one or more RCNNsare ensembled with non-max suppression and/or weighted box clustering.

Optionally, the step of thresholding the average malignancy outputvalues comprises selecting multiple operating points of the mammogram,optionally, selecting at least six operating points.

According to a second aspect, there is provided a method of training oneor more convolutional neural networks to perform the steps of anypreceding claim, the method comprising: receiving one or moremammograms; training one or more convolutional neural networks to whollyanalyse the one or more mammograms and determining a malignancy value;freeze the weights for the one or more convolutional neural networks;add RCNNs to the last convolutional layer of the one or moreconvolutional neural network; and train the RCNNs using the frozenweights of the one or more convolutional neural networks.

Optionally, the RCNNs comprise mask-RCNN heads.

Optionally, the one or more mammograms are restricted to 4000×4000pixels.

Optionally, the mammograms are pre-processed using any one or acombination of: windowing; resampling; and normalization.

According to a third aspect, there is provided an apparatus operable toperform the method of any preceding feature.

According to a fourth aspect, there is provided a system operable toperform the method of any preceding feature.

According to a fifth aspect, there is provided a computer programproduct operable to perform the method of any preceding feature.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will now be described, by way of example only and withreference to the accompanying drawings having like-reference numerals,in which:

FIG. 1 illustrates a flowchart showing an outline of the method of anembodiment;

FIG. 2 illustrates the portion of the flowchart of FIG. 1 focused onproviding a malignancy output based on the input image and thepre-trained malignancy detection neural network, optionally showing thepre-processing that can be applied to the input image;

FIG. 3 illustrates the Mask-RCNN of the embodiment of FIG. 1 in moredetail;

FIG. 4 illustrates the portion of the flowchart of FIG. 1 showing theprocess of the mean and max operations performed by the embodiment; and

FIG. 5 illustrates how the final output of the embodiment of FIG. 1 isdetermined.

SPECIFIC DESCRIPTION

FIG. 1 depicts an example embodiment which will now be described in moredetail below with reference to FIGS. 2 to 5 as appropriate.

Referring first to FIG. 1, there is shown a method for receiving inputmammography images 10 and outputting a malignancy output, for example ayes/no binary output or a more detailed output showing regions ofinterest along with a binary output.

In a medical scan of a patient (mammography), the scanned images arecollated in DICOM format, which is a file format commonly used to storemedical images. The method uses pre-processed data that is stored on aPicture Archiving Communication Systems (PACS) that radiologydepartments use in hospitals. The output of this method also enrichesthe PACS database to improve future applications of analysingmammographic images.

In some instances, the images can be pre-processed 20 using a variety ofmethods, including but not restricted to, windowing, resampling andnormalisation. The input images may also undergo domain adaption and/orstyle transfer techniques to further improve the results.

The mammograms, pre-processed or not, are then fed into a convolutionalneural network (CNN) classifier 30 which has been trained to analyse theimages and assess whether the image shows a malignant lesion. In someembodiments, there is use of more than one trained CNN to complete thistask. Conventional methods of detected malignant lesions in a mammogrammay also be used.

In order for a CNN to operate as a malignancy model the network firstneeds to be trained. Similar to the pre-processing methods mentionedabove, input images for the purpose of training the network may undergowindowing, resampling, normalisation, etc., before the images are used.In some instances, the images used to train the network are eitherprovided or sized to up to 4000×4000 pixels.

As the images are fed through the CNN, a number of stacked mathematicaloperations are performed. In doing so, the CNN applies variable tensorsto the previous layer such that a malignant or not score is produced asa result of these operations. We then update the variables based on thegradient of the cost function (cross-entropy) making use of thechain-rule to work out the gradient updates to apply. In this way,multiple CNNs can be trained to be used with the describedaspects/embodiments.

Additionally, the training of the CNNs may include concatenating aprevious image taken of the same mammographic view and run it throughthe networks together with the current image being fed into the network.This enables the fine tuning of the final few layers of the CNN suchthat they can account for multiple images.

Once the malignancy model(s) are trained, the network and its weightsare frozen. We then take one of the convolutional layer's outputs whichis then feed into mask heads from a Mask RCNN 40. An exemplary Mask RCNNis illustrated in FIG. 3. These heads include a bounding box predictor41, where the bounding boxes can be used to cut out a part of theoriginal image. In addition to, or on top of the cut-out patch, amalignant classifier 42 and segmentation 43 heads are placed. As withthe malignancy model, any conventional bounding box, malignancyclassifier or segmentation models can be used with this system. In “Maskr-cnn.” Computer Vision (ICCV), 2017 IEEE International Conference on.IEEE, 2017, He, Kaiming, et al. describe a traditional RCNN that can beused in at least some embodiments, which is incorporated by reference.

There are various methods of training the RCNNs. Firstly, connecting themalignancy model to the Mask RCNN the Mask RCNN heads can be trained atthe same time as the whole image malignancy model. Secondly, it is alsopossible to train the Mask RCNN without freezing the malignancy modelnetwork. Finally, the Mask RCNN heads may be trained with multiplemalignancy models. Thus, the method of training the Mask RCNN heads isnot restricted to a certain type, which enables the approach to betailored for specific uses.

Once the neural networks are trained, during use, or at inference time,the malignancy model is frozen based on the training data.

As an example, during run time, the system of the embodiment receivesfour types of mammography images: left cranial caudal view (L-CC) 51,right cranial caudal view (R-CC) 53, left medio-lateral-oblique (L-MLO)52 and a right medio-lateral-oblique (R-MLO) 54. This combination ofimages is known to be referred to as a case. Upon passing though themalignancy model or models, the system of the embodiment produces anentire case of outputs. These outputs are then averaged to generate asingle output 60Y.

As seen in FIG. 4, 51 represents an average score of all left cranialcaudal views, 52 represents an average score of all leftmedio-lateral-oblique (L-MLO) views, 53 represents an average score ofall right cranial caudal (R-CC) views and 54 represents an average scoreof all right medio-lateral-oblique (R-MLO) views. As depicted by 61 aand 62 a, the system of the embodiment then calculates a mean of therespective left side views 61 and right side views 62. This results in amalignancy output for each side. A max operation 63 is then performedfor the average malignancy outputs for each side.

Although not depicted in the figures, in the described embodiment themethod then thresholds this result with a predetermined threshold whichgives a binary malignant or not score 60Y.

Finally, with reference to FIG. 5, the score 60Y is used to gate whetheror not to show the Mask RCNN segmentations or bounding boxes 40X. Inthis way, instead of showing absolutely all lesions detected by the MaskRCNN alone, which leads to numerous false-positives, the resultin. MaskR-CNN outputs are only shown if the binary malignant score is positive,i.e. indicating malignancy. When 60Y does not indicate the case to bemalignant, the Mask RCNN outputs are ignored and no localisation data isproduced as an output of the system.

In some cases, the Mask RCNN results can be ensembled by interpolatingbetween bounding box coordinates (of shape [N, M, x1, x2, y1, y2] whereN represents the number of models and M the maximum number of boundingboxes) which have a sufficient intersection over union (IOU), which ispredetermined. Any bounding box which does not have a sufficient IOUwith the others are removed from consideration. With the resultingbounding boxes, the raw segmentation masks are then averaged beforethresholding with a predetermined threshold, and also averaging thelesion scores for all of the sufficient bounding boxes.

These operations result in a final set of bounding boxes of shape [1, M,x1, x2, y1, y2] along with a segmentation mask of shape [1, H, W] andlesion scores of shape [1, M]. A better way is to use weighted boxclustering (WBC) which is described by Paul F. Jaeger et al in “RetinaU-Net: Embarrassingly Simple Exploitation of Segmentation Supervisionfor Medical Object Detection” (https://arxiv.org/pdf/1811.08661.pdf),which is incorporated by reference.

As aforementioned, double reading is the gold standard in breast cancerscreening with mammography. In this scenario, two radiologists willreport on a case. Arbitration will occur when the two readers are not inagreement about whether to recall a patient for further screening tests.

In the present embodiment, the described system is able to operate as anindependent second reader. In the past, computer aided diagnosis systemswere not able to act as such due to a high false positive rate. Similarto a human radiologist, the described system of the embodiment can havea low false positive rate which means it can be used in at least thefollowing two ways:

-   -   1. As a truly independent second reader: a first (human)        radiologist looks at the case and the present system        independently assesses the case. If the two disagree, the system        of the embodiment shows the outlines for lesions of interest for        the human radiologist to consider, and if they agree, the        radiologist does not see the outputs of the system; or    -   2. As a non-independent second reader where the human        radiologist and the system of the embodiment both analyse the        case—in that the human radiologist is supported by the system of        the embodiment. The radiologist can click to see the results        generated by the system of the embodiment whenever they want.

Many approaches that mimic the techniques used by human radiologists canbe incorporated in the system in some embodiments, such as using aprevious image as a reference to look for any changes since the lastscan and also a mean then max operator to mimic the way humanradiologists trade off calling back a case.

Machine learning is the field of study where a computer or computerslearn to perform classes of tasks using the feedback generated from theexperience or data gathered that the machine learning process acquiresduring computer performance of those tasks.

Typically, machine learning can be broadly classed as supervised andunsupervised approaches, although there are particular approaches suchas reinforcement learning and semi-supervised learning which havespecial rules, techniques and/or approaches. Supervised machine learningis concerned with a computer learning one or more rules or functions tomap between example inputs and desired outputs as predetermined by anoperator or programmer, usually where a data set containing the inputsis labelled.

Unsupervised learning is concerned with determining a structure forinput data, for example when performing pattern recognition, andtypically uses unlabelled data sets. Reinforcement learning is concernedwith enabling a computer or computers to interact with a dynamicenvironment, for example when playing a game or driving a vehicle.

Various hybrids of these categories are possible, such as“semi-supervised” machine learning where a training data set has onlybeen partially labelled. For unsupervised machine learning, there is arange of possible applications such as, for example, the application ofcomputer vision techniques to image processing or video enhancement.Unsupervised machine learning is typically applied to solve problemswhere an unknown data structure might be present in the data. As thedata is unlabelled, the machine learning process is required to operateto identify implicit relationships between the data for example byderiving a clustering metric based on internally derived information.For example, an unsupervised learning technique can be used to reducethe dimensionality of a data set and attempt to identify and modelrelationships between clusters in the data set, and can for examplegenerate measures of cluster membership or identify hubs or nodes in orbetween clusters (for example using a technique referred to as weightedcorrelation network analysis, which can be applied to high-dimensionaldata sets, or using k-means clustering to cluster data by a measure ofthe Euclidean distance between each datum).

Semi-supervised learning is typically applied to solve problems wherethere is a partially labelled data set, for example where only a subsetof the data is labelled. Semi-supervised machine learning makes use ofexternally provided labels and objective functions as well as anyimplicit data relationships. When initially configuring a machinelearning system, particularly when using a supervised machine learningapproach, the machine learning algorithm can be provided with sometraining data or a set of training examples, in which each example istypically a pair of an input signal/vector and a desired output value,label (or classification) or signal. The machine learning algorithmanalyses the training data and produces a generalised function that canbe used with unseen data sets to produce desired output values orsignals for the unseen input vectors/signals. The user needs to decidewhat type of data is to be used as the training data, and to prepare arepresentative real-world set of data. The user must however take careto ensure that the training data contains enough information toaccurately predict desired output values without providing too manyfeatures (which can result in too many dimensions being considered bythe machine learning process during training, and could also mean thatthe machine learning process does not converge to good solutions for allor specific examples). The user must also determine the desiredstructure of the learned or generalised function, for example whether touse support vector machines or decision trees.

The use of unsupervised or semi-supervised machine learning approachesare sometimes used when labelled data is not readily available, or wherethe system generates new labelled data from unknown data given someinitial seed labels.

Machine learning may be performed through the use of one or more of: anon-linear hierarchical algorithm; neural network; convolutional neuralnetwork; recurrent neural network; long short-term memory network;multi-dimensional convolutional network; a memory network; fullyconvolutional network or a gated recurrent network allows a flexibleapproach when generating the predicted block of visual data. The use ofan algorithm with a memory unit such as a long short-term memory network(LSTM), a memory network or a gated recurrent network can keep the stateof the predicted blocks from motion compensation processes performed onthe same original input frame. The use of these networks can improvecomputational efficiency and also improve temporal consistency in themotion compensation process across a number of frames, as the algorithmmaintains some sort of state or memory of the changes in motion. Thiscan additionally result in a reduction of error rates.

Developing a machine learning system typically consists of two stages:(1) training and (2) production. During the training the parameters ofthe machine learning model are iteratively changed to optimise aparticular learning objective, known as the objective function or theloss. Once the model is trained, it can be used in production, where themodel takes in an input and produces an output using the trainedparameters.

Any system feature as described herein may also be provided as a methodfeature, and vice versa. As used herein, means plus function featuresmay be expressed alternatively in terms of their correspondingstructure.

Any feature in one aspect may be applied to other aspects, in anyappropriate combination. In particular, method aspects may be applied tosystem aspects, and vice versa. Furthermore, any, some and/or allfeatures in one aspect can be applied to any, some and/or all featuresin any other aspect, in any appropriate combination.

It should also be appreciated that particular combinations of thevarious features described and defined in any aspects can be implementedand/or supplied and/or used independently.

1. A computer-aided method of analysing mammographic images, the methodcomprising the steps of: receiving a plurality of mammograms; performinga first analysis on the plurality of mammograms comprising identifying amalignancy classification for each of the mammograms; determining amalignancy output value for each of mammograms dependent upon the firstanalysis; determining an average malignancy output value by averagingthe malignancy output values for the plurality of mammograms;thresholding the average malignancy output value to generate an outputbinary malignancy value performing a second analysis using a maskregional convolutional neural network (mask RCNN) on the plurality ofmammograms to determine a plurality of localisation data parameters foreach mammogram; and generating output localisation data for theplurality of mammograms in dependence upon the output binary malignancyvalue, wherein the second analysis comprises: generating a set ofbounding boxes, wherein generating the set of bounding boxes comprisesremoving any bounding boxes not having a predetermined threshold ofintersection over union and interpolating between bounding boxcoordinates; generating averaged lesion scores, wherein generating theaveraged lesion scores comprises averaging the lesion scores for the setof bounding boxes; and generating an averaged segmentation mask, whereinthe averaged segmentation mask is generated by averaging a plurality ofraw segmentation masks, before thresholding with a predeterminedthreshold.
 2. The method of claim 1 further comprising the step ofpre-processing to plurality of mammograms to improve malignancyclassification for each of the mammograms, the step of pre-processingfurther comprising the use of one or more trained neural networks. 3.The method of claim 1 wherein, the step of performing the first analysison the plurality of mammograms is conducted using one or more trainedconvolutional neural network classifier.
 4. The method of claim 3wherein the weights of the trained convolutional neural networkclassifier are frozen in dependence upon data used to train theconvolutional neural network classifier.
 5. The method of claim 1wherein, the plurality of mammograms comprises: a left side cranialcaudal mammogram, L-CC; a right side cranial caudal mammogram, R-CC; aleft side medio-lateral-oblique mammogram, L-MLO; and a right sidemedio-lateral-oblique mammogram, R-MLO.
 6. The method of claim 5wherein, the average malignancy output value comprises any combinationof; an average value for all L-CC malignancy output values; an averagevalue for all R-CC malignancy output values; an average value for allL-MLO malignancy output values; an average value for all R-MLOmalignancy output values; an average value for all left-side mammogrammalignancy output values; and an average value for all right-sidemammogram malignancy output values.
 7. The method of claim 6, wherein amax operator is performed between the average value for all left-sidemammogram malignancy output values and the average value for allright-side mammogram malignancy output values to determine the averagemalignancy output value.
 8. (canceled)
 9. The method of claim 1, whereinthe RCNN is a trained RCNN and comprises a plurality of sub-divisionalnetworks to determine the plurality of localisation data parameters, thesub-divisional networks provide any one or combination of: a boundingbox generation model; a segmentation model; and a malignancyclassification type model.
 10. The method of claim 9 wherein the RCNN iscoupled to an output convolutional layer of the one or moreconvolutional neural network used to perform the first analysis.
 11. Themethod of claim 10 wherein the RCNN is trained using the weights of theone or more convolutional neural network used to perform the firstanalysis.
 12. The method of claim 9 wherein the RCNN generates anoverlay mask indicating a lesion of interest, the mask further comprisesa malignancy probability value.
 13. The method of claim 9 wherein thebounding box generation model generates bounding box regression withnone-max suppression in order to locate lesions of interest.
 14. Themethod of claim 9 wherein the segmentation model provides a segmentationoutlines of anatomical regions and/or lesions, the segmentation modelfurther comprises localisational characteristics.
 15. The method ofclaim 9 wherein the malignancy classification type model identifies atissue type and density category classification for the breast.
 16. Themethod of claim 9 wherein the sub-divisional networks comprise anensemble of masks created by the sub-division networks.
 17. The methodof claim 16 wherein RCNN is ensembled with non-max suppression and/orweighted box clustering.
 18. The method of claim 1 wherein the step ofthresholding the average malignancy output values comprises selectingmultiple operating points of the mammogram, optionally, selecting atleast six operating points.
 19. A method of training one or moreconvolutional neural networks to perform the steps according to claim 1,the method comprising: receiving one or more mammograms; training one ormore convolutional neural networks to wholly analyse the one or moremammograms and determining a malignancy value; freeze the weights forthe one or more convolutional neural networks; add RCNNs to the lastconvolutional layer of the one or more convolutional neural network; andtrain the RCNNs using the frozen weights of the one or moreconvolutional neural networks.
 20. (canceled)
 21. (canceled)
 22. Themethod of claim 19 wherein the mammograms are pre-processed using anyone or a combination of: windowing; resampling; and normalization. 23.(canceled)
 24. (canceled)
 25. A computer program product operable toperform the method of claim 1.